The use of mobile devices, such as smart phones, wearable devices, tablets, laptops, etc., has been increasing. By understanding the content viewed by and actions taken by a user, the mobile device can build a much better user experience, for example by offering personalized predictions and assistance to the user. Part of understanding the content and actions involves identifying and classifying entities recognized in the content of the device screen. The entities may exist in an entity repository, such as a knowledge base or vision model, which stores facts and information about entities. Large, public entity repositories may include millions of such entities. Most computing devices, especially mobile computing devices, such as smart phones and tablets, have limited storage and use the entity repository via a connection with a server.